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ABOUT CHEA 


A national advocate and institutional voice for academic quality through 
accreditation, the Council for Higher Education Association (CHEA) is an 
association of 3,000 degree-granting colleges and universities and recognizes 
approximately 60 institutional and programmatic accrediting organizations in the 
United States. The CHEA International Quality Group (CIQG) serves as a U.S.- 
based international forum for quality assurance and accreditation. 
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ABOUT NORC 


NORC at the University of Chicago is an objective, non-partisan research institution 
that delivers reliable data and rigorous analysis to guide critical programmatic, 
business, and policy decisions. Since 1941, NORC has conducted groundbreaking 
studies, created and applied innovative methods and tools, and advanced principles 
of scientific integrity and collaboration. NORC conducts research in five main areas: 
Economics, Markets, and the Workforce; Education, Training, and Learning; Global 
Development; Health and Well-Being; and Society, Media, and Public Affairs. 
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INTRODUCTION 


The purpose of this report is to profile a selection of emerging approaches to judg- 
ing quality in higher education that represent a range of stakeholder interests and 
perspectives. This report includes three sections: 


Part 1. Quality Assurance at a Crossroads highlights perceived challeng- 
es for the U.S. accreditation system in a shifting higher education landscape 
and why this topic is of critical importance. 


Part 2. New Models for Judging Quality in Higher Education features 
new approaches that have emerged as potential complements to or replace- 
ments for the existing system of accreditation. First we describe models that 
are currently in practice or that include detailed proposals for implementa- 
tion, and then we summarize other noteworthy proposals under develop- 
ment. 


Part 3. Comparative Data Sets and National Rankings as Forms of 
Quality Review addresses the potential for comparative data sets to serve 
as a form of quality review. 


PART 1. QUALITY ASSURANCE AT A CROSSROADS 


For at least the last 60 years, accrediting organizations were considered the reliable 
authorities for assuring quality in U.S. colleges and universities (Eaton, 2015; Har- 
cleroad, 2011). Supporters of the current system consider the flexibility of the U.S. 
system of accreditation that allows institutions to set their own mission-driven goals 
and measures appropriately responsive to the diversity of institutions and programs 
that comprise it (Brown, Kurzweil, & Pritchett, 2017). Furthermore, supporters laud 
the nongovernmental structure of traditional accreditation and consider necessary 
the peer review processes that leverage relevant faculty and administrator expertise 
for assessing educational quality and fostering institutional improvement (Eaton, 
2015; Government Accountability Office [GAO], 2018). However, some observers 
perceive these same characteristics as key limitations of a system built for a by- 
gone era. 


As U.S. higher education transforms from an elite to a mass system—expanding 
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opportunity to new types of students, delivering content in new ways, and match- 
ing the pace of rapidly changing labor markets with new programs—some ques- 
tion the relevance and nimbleness of traditional quality review approaches. Are the 
foundational values and beliefs of U.S. accreditation such as institutional autonomy, 
academic freedom, and decentralization compatible with demands for uniform 
standards and public accountability that come with increasing investments in higher 
education? Can a system built for traditional institutions and programs and carried 
out by academics and administrators from the same be expected to assure the 
quality of an increasing array of non-traditional education and training providers? 
Exhibit 1 summarizes other selected! higher education changes underway, a num- 
ber of which traditional accreditation was not originally intended to address. 


Exhibit 1. Major Changes Underway in Higher Education 


Increasing Retention and 
diversity of New providers New credentials completion 
learners challenges 


Increasing costs 
and debt 


Changing Concerns about 


workforce Inequitable value and return 
outcomes 5 data 
demands on investment 


Waning public 
trust 


Proliferation of 


New approaches Increasing Increasing Potential conflicts 
(e.g., New finance Scrutiny of concern for of interest in 
competency- models accreditation consumer traditional peer 
based education) decisions protection review 


Considering today’s fast-evolving higher education context, experts note several 
specific aspects of accreditation that may be out of sync with these changes, includ- 
ing: (a) lack of attention to alternative providers; (b) conflicting roles and responsi- 
bilities for accreditors; (c) scope, definitions and measures of quality; and (d) lack of 
attention to public accountability and transparency (Ewell, 2015; GAO, 2018; Miller, 
Bergeron, & Martin, 2016). Each of these perceived challenges for accreditation is 
summarized briefly in the next section. 


1 More extensive discussions of the many groundbreaking changes underway in higher education are available elsewhere. See, for example, 
American Council on Education (2012), Ewell (2015), and Horn and Kelly (2015). 
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Accreditation Challenge #1: Alternative Providers of Higher Education 


Spurred by the multiyear economic slowdown of the last decade combined with 
rapid advances in technology, new and innovative education and career training 
models have emerged outside of traditional colleges and universities. These mod- 
els, which range from intensive place-based programs like coding bootcamps to 
nanodegrees developed by leading massive open online courses (MOOCs) provid- 
ers, have seen tremendous growth in recent years (Council for Higher Education 
Accreditation [CHEA], 2014; Horn & Kelly, 2015). For example, Class Central—a 
search engine and reviews site for MOOCs—reports that, as of 2017, an estimated 
81 million students worldwide have enrolled in at least one of the more than 9,400 
available MOOCs, as depicted in Exhibit 2 (Shah, 2017). Because many of these 
programs fall outside of the traditional system of accreditation, U.S. students want- 
ing to access these programs face two key barriers: (a) unaccredited educational 
providers are ineligible to participate in federal financial aid programs, and (b) no 
formal external reviews of quality are currently required of them (U.S. Department 
of Education [USDE], 2016). 


Exhibit 2. Major Changes Underway in Higher Education 


Growth of MOOCs: Students and courses Top 5 MOOC providers 
in 2017 by registered users 
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Note. Adapted from Shah (2013, 2014, 2016, 2016, 2017). 
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Accreditation Challenge #2: Conflicting Roles and Responsibilities for 
Accreditors 


Several reports address the inherent tensions in the current roles and responsibili- 
ties of accreditors that may limit the effectiveness of their efforts to oversee quality 
in higher education (Cooper, 2016; GAO, 2018; Miller et al., 2016). Some research- 
ers argue that the financial aid gatekeeping function that has come to define pres- 
ent-day conceptions of accreditation is in conflict with the self-regulation or quality 
improvement functions that originally characterized accreditation (Cooper, 2016; 
Ewell, 2008). Others criticize the accreditation “business model” for creating poten- 
tial financial conflicts of interest because accreditors’ main sources of revenue are 
dues and fees from the very institutions they oversee (GAO, 2018). 


Accreditation Challenge #3: Scope, Definitions and Measures of Quality 


Critics of accreditation argue that current quality review processes are too broad in 
scope and too focused on input criteria, such as facilities, faculty qualifications, and 
student support services, and not focused enough on critical outcomes like student 
learning, completion, and job placement (ACE, 2012; Kelly, James, & Columbus, 
2015; Miller et al., 2016). Moreover, because U.S. accreditors are a diverse collec- 
tion of independent organizations rather than a unified system of regulators, the 
existing structure is not designed to develop or promote “consensus standards” 
typical in other sectors. The resulting absence of common definitions and measures 
of quality is also the subject of growing criticism (GAO, 2018). 


Accreditation Challenge #4: Public Accountability and Transparency 


The U.S. accreditation system is increasingly called on to shift its central focus 

from quality improvement to public accountability, particularly given mounting costs 
of—and massive public and private investments in—higher education. However, the 
accreditation system was not originally created to directly serve the public interest 
and therefore may not be sufficiently transparent or helpful to students and other 
members of the public seeking practical information about academic quality (Ea- 
ton, 2011; Ewell, 2008; GAO, 2018). As noted in the previous section, this issue is 
further complicated by definitional and measurement differences across accrediting 
organizations. 


These critical challenges place immense pressure on the current system of accredi- 
tation to reform or make room for new entrants who will provide new approaches 
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tailored to meet these challenges. The next section presents some of the emerging 
approaches to judging quality in higher education that are seeking to fill this void. 


PART 2. NEW MODELS FOR JUDGING QUALITY IN 
HIGHER EDUCATION 


In recent years, numerous task forces, forums, expert panels, think tanks, policy- 
makers, and higher education providers have attempted to address the perceived 
limitations of traditional accreditation by proposing ideas, introducing legislation 

and piloting new models for judging quality in higher education. This section of the 
report presents selected profiles of new approaches to quality review that reflect a 
diversity of perspectives and interests. First, we describe the key features of models 
that have been piloted in the field or that offer detailed proposals for implementa- 
tion. Second, we review other proposals that are less well developed but still note- 
worthy. 


Piloted Models and Detailed Proposals 


For each of the following approaches, we begin by outlining the perceived 
challenge(s) for traditional accreditation that the approach aims to address, and 
then describe the approach in terms of five key features: 


1. Educational context refers to the context(s) for which the quality assurance 
approach is proposed or currently carried out. 


2. Quality definitions focuses on the particular definition(s) or measure(s) of 
quality the approach advocates. 


3. Who defines quality describes the actors responsible for setting the stan- 
dards by which quality is or should be assessed. 


4. Who assesses quality describes the actors responsible for determining com- 
pliance with the standards. 


5. Role of money addresses whether and how money factors into the ap- 
proach. 
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EQUIP Quality Assurance Entities (QAEs) 


Introduced in 2015, the U.S. Department of Education’s (USDE) Educational Qual- 
ity through Innovative Partnerships (EQUIP) experiment is intended to encourage 
increased innovation in higher education through partnerships between Title IV- 
eligible colleges and universities and Title IV-ineligible nontraditional education and 
training providers. EQUIP tests new ways of promoting access to innovative educa- 
tion and training opportunities that fall outside the current financial aid system. This 
program allows institutions to award federal financial aid to eligible students en- 
rolled in selected programs for which 50 percent or more of the content and instruc- 
tion is provided by nontraditional providers (USDE, 2016). EQUIP aims to advance 
quality assurance processes that focus on student learning and other outcomes 
through the use of independent Quality Assurance Entities (QAEs) (USDE 2015). 
The third-party QAEs establish and carry out rigorous quality assurance process- 
es—establishing standards and reviewing, monitoring and reporting on programs 
and providers according to USDE’s guidance. Exhibit 3 details the key features of 
the EQUIP QAEs. Exhibit 4 summarizes the eight original EQUIP pilot partnerships 
selected in 2016 and their current status. Of note, only five partnerships remain in 
the pilot as of 2018 (McKenzie, 2018). 


Mis» 
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Exhibit 3. Key Features of EQUIP Quality Assurance Entities* 


Quality Assurance Entities (USDE) 


Accreditation Challenges Explicitly Addressed 


Vv Alternative Providers of Higher Education ” Scope, Definitions and Measures of Quality 
Y Conflicting Roles and Responsibilities for 
Accreditors 


EQUIP partnerships pair Institutions of Higher Education and 
alternative education providers. 


Educational Context 


* EQUIP is focused on low-cost, short-term programs and low-income students. 
* Nontraditional partners include coding bootcamps, online course providers and employer organizations. 


Quality Definitions Definitions and measures focus on outcomes. 


* Outcomes relate to the USDE guiding questions and address (a) claims for learning, (b) assessment and 
student work, (c) outputs and (d) program management. 


UTS TSN TSTNENN | QAEs set quality standards specific to their EQUIP partnership. 


* QAE-defined standards must be consistent with the USDE guiding questions. 
* QAEs also establish quality assurance processes for reviewing their programs and providers. 


Wo Aseses ty | iCoecees elle.) QAEs carry out quality reviews and monitor programs and providers. 


* QAEs monitor programs at regular intervals reflecting their program's student enrollment trajectories. 
* QAEs report findings to institutions, accreditors and USDE every six months or when a concer arises. 


Role of Money Students access federal student aid to enroll in approved programs. 


« Accredited colleges and universities receive waivers from USDE that allow them to award federal financial 
aid to students enrolled in programs primarily offered through the unaccredited partner provider. 


*USDE=USS. Department of Education, EQUIP=Educational Quality through Innovative Partnerships, IHE=Institution of Higher Education, 
QAE=Quality Assurance Entity. 
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Exhibit 4. Selected EQUIP Sites and 2018 Status* 


Institution: Colorado State 
University Global Campus 
(Greenwood Village, CO) 


Nontraditional 
provider: Guild Education 


QAE: Tyton Partners 


Status: Withdrawn/On 
Ho 


Institution: Dallas Community 
College System (Dallas, TX) 


Nontraditional 
provider: StraighterLine 


QAE: CHEA Quality 
Platform 


Institution: Marylhurst 
University (Marylhurst, OR) 


Nontraditional 
provider: Epicodus 


QAE: Climb 


Status: Pending Approval 


Institution: Northeastern 
University (Boston, MA) 


Nontraditional 
provider: General Electric 


QAE: American Council on 
Education 


Institution: University of Texas- 
Austin (Austin, TX) 


Institution: SUNY Empire State 
College (Saratoga Springs, NY) 
Nontraditional 
provider: MakerSquare 


Nontraditional 
provider: The Flatiron 
School 


QAE: American National 
Standards Institute 
Status: Withdrawn/On 
Status: Pending Approval Hold 


Institution: Wilmington 
University (New Castle, DE) 


QAE: Entangled Solutions 


and Moody, Famiglietti & 
Andronico, LLP 


Institution: Thomas Edison State 
University (Trenton, NJ) 


Nontraditional Nontraditional 


provider: Zip Code 
Wilmington 


QAE: HackerRank 


provider: Study.com 


eee i ae nee 


Status: Pending Approval 


Status: Withdrawn/On 
Hold 


Status: Pending Approval 


*CHEA = Council for Higher Education Accreditation, EQUIP = Educational Quality through Innovative Partnerships, 
QAE = Quality Assurance Entity. Adapted from U.S. Department of Education (2016) and McKenzie (2018). 


Status: Approved 


CHEA Quality Platform 


The Council for Higher Education Accreditation’s (CHEA) Quality Platform is intend- 
ed to address the growth of alternative providers of higher education. The Quality 
Platform establishes outcomes-based standards for conducting voluntary external 
quality reviews of alternative providers with a core focus on student achievement. 
Quality Platform standards are suitable for use in the United States and interna- 
tionally and reviews can be carried out by a range of entities, including existing 
accreditation or quality assurance bodies, higher education associations and other 
organizations created expressly to conduct such a review. CHEA piloted the Quality 
Platform with the DeTao Masters Academy Advanced Classes, a private company 
in China offering nontraditional education programs. In addition, CHEA currently 
employs the approach in its role as a QAE for an EQUIP partnership between the 
Dallas County Community College District and StraighterLine, a U.S. company that 
offers low-cost, online higher education courses (CHEA, 2016). Of the original eight 
pilot projects selected to participate in the EQUIP program, this partnership is the 
first to receive final approval from the Department of Education (McKenzie, 2018). 
Exhibit 5 details the key features of the CHEA Quality Platform. 
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Exhibit 5. Key Features of the CHEA Quality Platform 


The Quality Platform (CHEA) 


Accreditation Challenges Explicitly Addressed 


Y Alternative Providers of Higher Education  “ Public Accountability and Transparency 
¥ Scope, Definitions, and Measures of 
Quality 


| Educational Context | Context This approach is tailored to alternative providers of higher education. 


* This approach is intended for a range of program types, including short-term courses, certificate- and 
badge-granting experiences, noncredit online career develpment modules, etc. 


| Quatity Definitions | Definitions Definitions and measures focus on outcomes. 


* The primary focus is student achievement (i.e., what students learn and can do). 
* Asecondary focus in defining quality is providers' capacity and resources, especially technology. 


| Who Defines Quality | Defines Quality Providers define quality within the Quality Platform outcomes framework. 


* The Quality Platform also relies on research and analysis, pilot testing, and consultation with the quality 
assurance, higher education, and employer communities in administering its approach. 


| Who Assesses Quality | ISS eltelisy | This approach includes self-review and external review. 


* Quality assurance bodies, higher education associations, and other third parties carry out voluntary 
external reviews roughly every three years. These reviews include a transparency requirement. 


| Role of Money —_ | of Money Providers pay a fee for the Quality Platform review. 


* Successful reviews result in the designation "Quality Platform Provider." Review findings provide students 
and the public with reliable information about provider performance that can inform their investment 
decisions. 


Center for American Progress Alternative Gatekeeping System 


The Center for American Progress’s (CAP’s) detailed proposal for an Alternative 
Gatekeeping System for granting access to federal financial aid is intended to 
foster innovation while protecting higher education consumers (Miller et al., 2016). 
CAP recommends its Alternative Gatekeeping System as a “complementary com- 
petitor” to the existing system of accreditation that addresses limitations related 

to conflicting roles and responsibilities, alternative providers, and scope and mea- 
sures (Miller et al., 2016, p.1). Similar to other new approaches to judging 
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quality, the Alternative Gatekeeping System proposes using private third parties (in- 
cluding existing accrediting organizations and new actors such as professional and 
academic membership organizations) to set outcomes-based quality standards and 
measures, with a specific focus on student outcomes and provider financial health. 
Third parties would also set minimum thresholds for their proposed outcome mea- 
sures that may vary based on provider or program type. Under this model, the fed- 
eral government would select and review third-party standard-setters, collect and 
verify provider data, and enforce penalties in case of noncompliance with standards 
(Miller et al, 2016). Exhibit 6 details the key features of the Alternative Gatekeeping 
System proposal. 


Exhibit 6. Key Features of the Alternative Gatekeeping System Proposal* 


Alternative Gatekeeping System (CAP) 


Accreditation Challenges Explicitly Addressed 


v Alternative Providers of Higher Education v Scope, Definitions and Measures of Quality 
Y Conflicting Roles and Responsibilities vy Public Accountability and Transparency 


| Educational Context | Context The system is suitable for any provider or program but has not been piloted. 


¢ The system is designed for educational providers and programs that want to participate in U.S. federal 
student financial aid programs. 


| Quality Definitions | Definitions Definitions and measures would focus on outcomes. 


* Student outcomes include completion, job placement, earnings of graduates and federal loan repayment. 
* Outcomes also address provider financial stability (e.g., ratios of assets to liabilities and debts to assets. 


| wh Defines Quay | Defines Quality Private third parties would set quality standards, measures and thresholds. 


¢ USDE would select and review standard-setters and approve and monitor the effectiveness of standards. 
* New actors other than traditional institutional accreditors could be standard-setters. 


| who Assesses Quality | Assesses Quality USDE would assess providers’ performance against quality standards. 


¢ USDE would collect and verify relevant performance data directly from providers. 
* USDE would enforce standards and determine providers’ financial aid eligibility. 


Role of Money of Money Students in approved programs could access federal student financial aid. 


* USDE would publicly release all decisions and related justifications for approval or termination of a 
provider's participation in federal financial aid program. 


* CAP = Center for American Progress, USDE = U.S. Department of Education. 
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Entangled Solutions Quality Assurance (QA) Standards 


Entangled Solutions, a company that focuses on innovation in higher education, is 
one of the QAEs selected to monitor student outcomes in an EQUIP partnership— 
though this partnership is currently on hold as shown previously in Exhibit 4. Entan- 
gled Solutions recently released its Quality Assurance (QA) Standards with a goal 
of creating unified standards for audited outcomes reporting. The QA Standards 
were developed with the help of a 25-member task force representing traditional ac- 
ademic institutions, coding bootcamps, professional services and investment firms, 
and policymakers. Entangled Solutions emphasizes transparency and accountabil- 
ity and recommends its approach as an alternative to traditional accreditation that 
is suitable for traditional and alternative providers of higher education (Entangled 
Solutions, 2018). Exhibit 7 details the key features of Entangled Solutions’ Quality 
Assurance Standards. 


Exhibit 7. Key Features of Entangles Solutions’ Quality Assurance Standards 


Quality Assurance Standards (Entangled Solutions) 


Accreditation Challenges Explicitly Addressed 


v Alternative Providers of Higher Education v Scope Definitions and Measures of Quality 
¥ Conflicting Roles and Responsibilities v Public Accountability and Transparency 


| Educational Context | Context The approach is suitable for traditional and alternative providers. 


* Entangled Solutions originally designed the Quality Assurance (QA) Standards for alternative providers, 
like coding bootcamps and online course providers, but now promotes the approach for any provider type. 


| Quality Definitions | Definitions Definitions and measures focus on outcomes. 


* QA Standards relate to learning, completion, satisfaction, job placement and earnings. 
*« Measures selected must support a provider's claims about its program outcomes and student success. 


| Whe Defines Quality Defines Quality Entangled Solutions and 25-member task force set initial standards. 


+ Entangled Solutions will set up a nonprofit organization to maintain the QA Standards and support the 
auditing process. 


| Who Assesses Quality Assesses Quality Third-party auditors carry out external quality review. 


¢ Independent auditors monitor compliance with QA Standards, but students, employers and funders make 
final judgements of quality based on publicly available audit findings. 


Role of Money The approach is designed to help students determine where to invest. 


¢ The third-party auditor will have no financial relationship to the provider being reviewed. 
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QA Commons Essential Employability Qualities Certification (EEQ CERT) 


The Quality Assurance Commons for Higher and Postsecondary Education (QA 
Commons) is a project of the National Center for Higher Education Management 
Systems funded by a grant from Lumina Foundation. The mission of the QA Com- 
mons is to develop and test alternative approaches to quality assurance that are 
responsive to the changing higher education landscape (QA Commons, 2018a). 
The core focus of the QA Commons is assuring students, families and employers 
that higher education programs prepare students for employment and success. The 
Essential Employability Qualities Certification (EEQ CERT) is QA Commons’ pro- 
gram-level approach to certifying baccalaureate and sub-baccalaureate programs 
through an employer-recognized quality review process (QA Commons, 2018b). 
Although many of the essential employability qualities that make up the EEQ CERT 
build on existing learning and outcomes frameworks (including Lumina Founda- 
tion’s Degree Qualifications Profile and the Association of American Colleges & 
Universities’ Essential Learning Outcomes), the main emphasis of the EEQ CERT 
is employment-related outcomes. Twenty-seven higher education programs at 14 
institutions are currently participating in a pilot to field test the approach’s proposed 
criteria (QA Commons, 2018c). Exhibit 8 details the key features of the EEQ CERT. 
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Exhibit 8. Key Features of the EEQ CERT* 


EEQ CERT (QA Commons) 


Accreditation Challenges Explicitly Addressed 


Quality 


| euctonl Cont feyiicyeae he approach currently focuses on traditional providers of higher education. 


* The current pilot includes 27 baccalaureate and sub-baccalaureate programs in a variety of disciplines 
within a variety of institution types. In all, 14 institutions are participating in the pilot. 


Quality Definitions Criteria for EEQ certification focus on workforce readiness outcomes. 


* Program certification is based on students’ knowledge, skills, abilities and experiences for employability. 
* Outcomes also focus on soft skills (¢.g., communication, problem-solving, collaboration adaptability, etc.) 


Wit eeiiestele\is7 | Academic experts in partnership with students and employers define criteria. 


+ Pilot programs serve as "co-designers” of the EEQ CERT approach. 
+ Employer representatives play a central role through two QA Commons advisory bodies. 


Who assesses Quy ESSE ee is) Academic experts, students and employers play a role in quality review. 


* EEQ certification will be awarded based on outcomes-based verifiable evidence. 
* The EEQ pilot will determine the best way to validate and evaluate data provided by programs. 


/ Role otMoney of Money To be determined. 


* The EEQ pilot seeks to determine how to align the EEQ certification approach with other quality 
assurance processes. 


*EEQ CERT=Essential Employability Qualities Certification, QA Commons=Quality Assurance Commons for Higher and Postsecondary 
Education 
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Council on Integrity in Results Reporting (CIRR) Standards 


Coding schools or coding bootcamps are an important example of alternative 
providers of higher education that have gained attention in recent years. Although 
these short-term intensive training programs are promoted as pathways to high- 
paying programming jobs in the digital economy, recent closings and growing 
complaints in the coding bootcamp industry have led to calls for uniform quality as- 
surance standards and processes for these providers (Lohr, 2017; McBride, 2016). 
The Council for Integrity in Results Reporting (CIRR) is a nonprofit code school 
membership organization that began as a project of Skills Fund, an organization 
that offers financing to code school students. CIRR defines and maintains qual- 

ity standards, requires members to annually assess and publicly release student 
outcomes reports; and requires members to obtain annual, third-party verification of 
their documentation and reporting (CIRR, 2018). Twenty members currently em- 
ploy CIRR standards, with many schools producing CIRR reports for multiple sites 
(CIRR, 2018). Exhibit 9 charts the growth of coding bootcamps in the United States 
and Canada since 2013 as well as CIRR member locations. Exhibit 10 details the 
key features of the CIRR approach. 


Exhibit 9. Growth of U.S. and Canadian Coding Bootcamps and CIRR Members* 


Enrollment in U.S. and : CIRR Member Coding 
25,000 Canadian Coding 9 io ath 
20,000 f: t 
fe sizies, Aye gil tiladelphia 
15,000 
4s Vegas 
10,000 zed he 
_ |_| U | % 
? *% Guilt of 
2013 2014 2015 2016 2017 : Mexico —_ 


“Adapted from Course Report (2017) and CIRR (2018). 
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Exhibit 10. Key Features of the Council on Integrity in Results Reporting 


Council on Integrity in Results Reporting (CIRR) Standards 


Accreditation Challenges Explicitly Addressed 


Y Alternative Providers of Higher Education Public Accountability and Transparency 


¥ Scope, Definitions, and Measures of 
Quality 


| Educational Context | Context The approach is designed for alternative providers of higher education. 


* The approch is specifically targeted at code schools and/or coding bootcamps and is currently used by 20 
schools, many of which have multiple sites. 


} uty Detntons | Definitions Definitions and measures focus on outcomes. 


* Definitions, measures, and reporting primarily focus on graduation and job placement outcomes (e.g., on- 
time graduation, full-time and part-time job placement, in- and out-of-field job placement, and salary). 


Who Decides Quality CIRR members set standards. 


+ Standards are intended to be transparent, easy to understand, and "student-first," reflecting what 
prospective students want to know about the coding schools’ outcomes before deciding to enroll. 


| Whe Assesses Quality | ECSeSSS (elieifiay Students and other stakeholders ultimately judge quality. 


* Member institutions generate their own CIRR standard report and obtain annual third-party verfication. 
* Verified reports must be publicly available for students and others to use to make quality judgments. 


} Role otMoney | of Money The approach aims to provide information to influence investment decisions. 


* Skills Fund, an alternative financing platform for students in coding bootcamps, uses CIRR standards to 
evaluate schools they work with. 


Other Noteworthy Proposals Addressing Quality Review 


This section provides brief descriptions of noteworthy proposals for alternative 
quality review approaches that are under development. These approaches include 
the U.S. Chamber of Commerce Foundation’s employer-led approach and several 


legislative proposals. 
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U.S. Chamber of Commerce Foundation Employer-Led Quality Review 


The U.S. Chamber of Commerce Foundation (2016) proposed two approaches 

for incorporating employers into processes for assuring quality in higher education 
intended to address the skills gaps of recent college graduates entering America’s 
businesses. The first approach calls for strengthening the employer voice in cur- 
rent accreditation structures to improve institutional responsiveness to employer 
needs (U.S. Chamber of Commerce Foundation, 2016). The second approach calls 
for an employer-led quality review system based on leading practices from supply 
chain management. This approach may allow institutions to become recognized as 
preferred human capital suppliers for employers. 


Higher Education Innovation Act (S.615, 2017) 


This bill was first introduced by Senators Michael Bennet (D-CO) and Marco Rubio 
(R-FL) in 2015 and seeks to amend Title IV of the Higher Education Act of 1965 to 
create an alternative outcomes-based process for allowing students to use federal 
financial aid to access high-quality, innovative and effective programs offered by 
colleges, universities and alternative higher education providers. The bill calls for 
the development of a five-year pilot program in which providers would be required 
to produce data demonstrating successful student outcomes. The bill also calls for 
“innovation authorizers,” designated by USDE, to define appropriate outcomes and 
measures and to authorize eligible entities to provide instruction and Title IV funds 
to students (S.615, 2017, p.2). 


Higher Education Reform and Opportunity Act (S.2228, 2017) 


This bill was introduced by Senator Mike Lee (R-UT) in 2017 and would allow 
states to create an alternative accreditation process for designating colleges, 
universities, nonprofit organizations, for-profit organizations or businesses and 
postsecondary apprenticeship programs as eligible for Title IV funding. It also al- 
lows states to administer federal financial aid for institutions and requires increased 
transparency about financial aid, alumni employment and federal loan default rates 
(S.2228, 2017). 


Accreditation Reform and Enhanced Accountability Act (S.3380, 2016) 


This bill was introduced by Senators Elizabeth Warren (D-MA), Brian Schatz (D-HI) 
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and Dick Durbin (D-IL) and calls for the U.S. Secretary of Education to establish cri- 
teria for accrediting agencies and associations to use for judging quality, including 
student achievement measures and federal loan repayment and default rates. This 
proposal would ban administrative officers and others with a financial interest in an 
institution from serving on the board or commission of that institution’s accreditor 
(S.3380, 2016). It would also provide for differentiated accreditation status such 

as “accredited with distinction” or “with risk.” This legislation creates penalties for 
accrediting agencies that fail to appropriately vet institutions found to have harmed 
students or engaged in abusive or deceptive practices. 


PART 3. COMPARATIVE DATA SETS AND NATIONAL 
RANKINGS AS FORMS OF QUALITY REVIEW 


The previous section presented a diverse selection of approaches to quality review. 
Some approaches wrestled with quality assurance challenges for alternative provid- 
ers of higher education, others aimed to improve transparency and public account- 
ability in the quality review process, and nearly all approaches stressed the use 

of standardized definitions and measures of quality, as well as the use of student 
outcomes rather than institutional and program-level inputs. Nearly all profiled 
approaches involved a third party to provide oversight or verification, or to carry 

out the quality review process. This last section features another potential type of 
quality review: comparative data. While the resources featured in this section were 
created as tools for reviewing institutional data and comparing colleges and univer- 
sities across the United States and not as quality review tools, there is some debate 
if they also can potentially be used as a form of quality review. 


The College Scorecard 


The College Scorecard (2015) provides students, families, guidance counselors, 
nonprofits and other key stakeholders with institutional data through an online tool 
that has been accessed by over 2.5 million users (Kreighbaum, 2017). Users can 
easily find the average annual cost of an institution, its graduation rate, the typical 
salary post-graduation, and information on student debt levels. 


This tool has some key strengths including data transparency and the ability to 
compare institutions, but it also has several key limitations that would need to be 
addressed in order to maximize its use as a quality review tool. 
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¢ The tool currently lacks data on student learning and preparing students to 
be active and engaged citizens. 


¢ Graduation rate data are included, but graduation rate data for part-time and 
transfer students (first released in 2017) are not included. 


¢ Median earnings data are included but are not disaggregated by major or 
adjusted to account for the cost of living where graduates live, and only covers 
federal financial aid recipients, which in some cases is a small subset of an 
institution’s overall student population (Council of Economic Advisers, 2017; 
Williams, 2018). 


Despite these limitations, the College Scorecard may be of some use in quality 
reviews, specifically for determining the average annual cost of an institution, its 
graduation rate, the typical salary post-graduation, and information on student debt 
levels. 


Accreditor Dashboards 


Building off the College Scorecard, which sought to increase transparency, en- 
hance accountability and promote increased understanding of student outcomes of 
institutions that are approved by accrediting agencies, the Obama administration 
released the Accreditor Dashboards in 2016. The dashboards use data from the 
College Scorecard and are intended to help inform the National Advisory Commit- 
tee on Institutional Quality and Integrity (NACIQI), which “provides recommenda- 
tions regarding accrediting agencies that monitor the academic quality of postsec- 
ondary institutions and educational programs for federal purposes” (USDE, 2018, 
p.1). The dashboards include data visualizations that demonstrate the performance 
of the schools associated with the different accrediting agencies (USDE, 2018). 
The different dashboards are separated by accreditor and display data on “institu- 
tional characteristics, cost and debt levels, and completion rates and post-college 
outcomes— including typical earnings and repayment outcomes—of all the colleges 
and universities they accredit” (USDE, 2018, p.1). 


The Accreditor Dashboards contain data in four key areas: 
1. The performance and outcomes of the institutions the agency accredits 


2. The evidence and data used in decision-making by the accrediting agency 
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3. The accrediting agency’s standards and practices for assessing student 
achievement 


4. The accrediting agency's activities aimed at improving program or institu- 
tional quality 


In order to inform NACIQI’s review process and to promote data transparency, ED 
created the accreditor dashboards. The data collected through this partnership are 
also data that could inform the arbiters of quality. 


Ranking Systems 


Collegiate ranking guides predate the College Scorecard and the Accreditor Dash- 
boards, and are an influential, multimillion dollar industry (McDonough, Antonio, 
Walpole, & Pérez, 1998). Institutional administrators pay close attention to popular 
ranking guides, including those from the U.S. News and World Report (USNWR) 
and the Princeton Review, because fluctuation in these rankings affects enrollment, 
alumni giving, student and faculty morale, and campus resources (Meredith, 2004). 
Despite the popularity of rankings like USNWR or the Princeton Review, though, 
the majority of incoming freshmen do not use them (McDonough et al., 1998). Stu- 
dents using these guides account for 40 percent of incoming freshman and tend to 
have a higher socioeconomic status (McDonough et al., 1998). The Education Trust 
(2010), found even more staggering figures: Dependent students from families 
making less than $30,000 per year only account for 20 percent of college students 
using these guides (only 13 percent at “public research extensive universities”) (p. 
2). It is believed that the majority are not using these ranking guides because the 
information being ranked is not particularly useful to the rest of the college-going 
population (McDonough et al., 1998). While ranking systems garner lots of excite- 
ment among the popular press and are used by many individuals exploring colleges 
and universities, more transparent tools for making institutional comparisons have 
emerged over the years (e.g., College Scorecard, Accreditor Dashboards) to poten- 
tially provide a more complete review of available data. 
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CONCLUSIONS 


The rapidly changing landscape of higher education is challenging the fundamental 
role, structure and scope of traditional, accreditation-based approaches to judging 
quality in higher education. The emerging approaches to quality assurance and 
review presented in this report offer new ways of defining and measuring quality, 
providing public accountability and transparency, addressing alternative providers 
of higher education, and identifying a mix-match in expectations of the roles and re- 
sponsibilities of accreditors. Ultimately, students and the public need reliable ways 
of comparing and contrasting providers so they can choose high-quality programs 
that are appropriate to their academic and occupational goals. The piloted and pro- 
posed approaches reviewed in this report take on this challenge in different ways. 


The models reviewed in this report focus on a wide array of quality outcomes, 
including student achievement, persistence, earnings, job placement, loan repay- 
ment, satisfaction, employability skills, and abilities. Many of the new definitions of 
quality represent a shift away from process evaluation and towards a focus on out- 
comes, diminishing much of the improvement function of traditional accreditation. 
These approaches are offered as complements, competitors, and in some cases 
replacements for the current array of national, regional and programmatic accredit- 
ing bodies and processes. The approaches also introduce new standard-setting 
and compliance roles for the federal government, the public, new third parties and 
state governments. The expanded roles for state and federal governments in some 
of these proposals may provide new opportunities for comparison across sectors 
normally accredited by different bodies, although government funding for these 
expanded roles may be a challenge. 


The deference and autonomy once granted to accrediting organizations to decide 
the value and fate of providers of higher education are waning in the face of the 
challenges posed by the needs of new types of students and innovative business 
and content delivery models. The existing system of accreditation will remain under 
pressure to reform and adapt to the evolving needs of students and providers. And 
adaptability will ensure accreditation continues to play its critical role in helping the 
public and students identify and invest in quality institutions and programs, as well 
as maintain a healthy and effective system of higher education in the United States. 
While alternative approaches to quality review like those detailed here can provide 
insights and avenues for supporting our evolving system and its stakeholders, 

we need to know more about how their implementation supports the quality and 
transparency goals of quality assurance in the current higher education landscape. 
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Learning from these innovations can create a postsecondary education system 
with new models of delivery and types of providers while still providing high-quality 
outcomes, transparency, and accountability to students, parents, policymakers, 
employers, the general public and other key stakeholders. “* 


= = ——————— 
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